Thanks @XiaoHongbo-Hope! A question on necessity first.

The direction of the read effort (#413) is to let pypaimon run its DataFrame read on the Rust core — initially as a basic, opt-in path behind a config flag, not a wholesale replacement, so it can mature alongside the pure-Python path. In that model Rust both plans and reads: PR3 already exposes new_read().read(splits) returning Arrow from the Rust TableRead, and splits stay opaque on the Python side — a transport token, nothing more — so pypaimon never needs to look inside one.

to_dict() exposes the full internal split contents, which is only needed if something rebuilds the split and reads the files outside Rust — i.e. reading in Python, which is the opposite of the direction #413 is moving in.

So: what's the use case for exposing split internals? If it's for a Rust-plans / Python-reads path, I think we should align on that direction first — otherwise it risks pulling us away from the opt-in Rust read path we're building toward.

Thanks @XiaoHongbo-Hope! A question on necessity first.

The direction of the read effort (#413) is to let pypaimon run its DataFrame read on the Rust core — initially as a basic, opt-in path behind a config flag, not a wholesale replacement, so it can mature alongside the pure-Python path. In that model Rust both plans and reads: PR3 already exposes new_read().read(splits) returning Arrow from the Rust TableRead, and splits stay opaque on the Python side — a transport token, nothing more — so pypaimon never needs to look inside one.

to_dict() exposes the full internal split contents, which is only needed if something rebuilds the split and reads the files outside Rust — i.e. reading in Python, which is the opposite of the direction #413 is moving in.

So: what's the use case for exposing split internals? If it's for a Rust-plans / Python-reads path, I think we should align on that direction first — otherwise it risks pulling us away from the opt-in Rust read path we're building toward.

Thanks, Junrui. Got your concern, will discuss with Jingsong and then back to you.

@JunRuiLee Thanks Junrui. My view: the read side can be handled by adding more workers in production, but plan runs on the driver. And It's additive and opt-in — no storage-format change — and it doesn't block #413; once full-Rust read lands, this just becomes an alternate/interim path.

Let's create an unified protocol: apache/paimon#8482